Skim

In order to compute summary statistics of our variables and to find NAs, we will now run both the favstats() and the skim() function.

skimr::skim(listings)

Variable type: character

Variable type: Date

Variable type: logical

Variable type: numeric

We find that the original data set includes: - 43 character variables - 41 numeric variables - 17 logical variables - 5 date variables

(before data cleaning done in previous section)

After the first data cleaning in the section above, we find:

• 35 character variables
• 49 numeric variables
• 17 logical variables
• 5 date variables

MISSING VALUES

This is how many missing values we find within our relevant variables:

• price: 0 missing

• extra_people: 0 missing

• property_type: 0 missing

• room_type: 0 missing

• number_of_reviews: 0 missing

• longitude , latitude: both 0 missing

• neighbourhood_cleansed: 0 missing

• cleaning_fee: 7,049 missing

• review_scores_rating: 7,314 missing

Note: We will go with price as relevant price variable as it indicates the price per night and the other price variables (weekly_price and monthly_price) have several missing values.

As a missing value for cleaning_fee highly likely indicates, that there is no cleaning fee, we will convert all NAs to 0.

We already changed the cleaning_fee data type to numeric in the previous section and will now substitute all NAs with 0s as described.

#replace all NAs with 0
listings <- listings %>%
  mutate(cleaning_fee = case_when(
    is.na(cleaning_fee) ~ 0, 
    TRUE ~ cleaning_fee
  ))

#skimr::skim(listings)
# find no more missing values for cleaning_fee

In order to still capture the observations that have a missing review_scores_rating but don’t skew the average scores rating, we will calculate the mean of all scores (except NAs) and then convert all NAs into that mean score.

#calculate the mean of all scores except NAs
avg_review_scores_rating <- listings %>% 
  # pull the review_scores_rating as a list
  pull(review_scores_rating) %>% 
  mean(na.rm = TRUE)

#convert NAs to the mean score
listings <- listings %>% 
  mutate(review_scores_rating = case_when(
    is.na(review_scores_rating) ~ avg_review_scores_rating,
    TRUE ~ review_scores_rating
  ))

#skimr::skim(listings)
# find no more missing values for review_scores_rating

Note: Variables with 100% missing values will be eliminated later)

PROPERTY TYPE

Looking at property_type, we can use the count function to determine how many categories there are their frequency.

#count property types
listings %>% 
  count(property_type) %>% 
  arrange(desc(n)) #arrange in descending order of count

We find that the top 4 property types are “Apartment” with 9,670 listings, “House” with 7,030 listings, “Guest suite” with 1,488 listings and “Villa” with 1,051 listings. Together these top 4 property types concern 19,239 listings, meaning they account for 80% (19,239 / 24,062) of all listings and are, therefore, most relevant for our analysis.

Since the vast majority of the observations in the data are one of the top four property types, we would like to create a simplified version of property_type variable that has 5 categories: the top four categories and Other.

#mutate to 5 types
listings <- listings %>%
  mutate(prop_type_simplified = case_when(
    property_type %in% c("Apartment","House", "Guest suite","Villa") ~ property_type, 
    TRUE ~ "Other"
  ))
 
#check prop_type_simplified compared to property_type
listings %>%
  count(property_type, prop_type_simplified) %>%
  arrange(desc(n))      

#check only prop_type_simplified
listings %>%
  count(prop_type_simplified) %>%
  arrange(desc(n))   

MINIMUM NIGHTS

As Airbnb is most commonly used for travel purposes, i.e., as an alternative to traditional hotels, we only want to include listings in our regression analysis that are intended for travel purposes.

#count values for minimum_nights
listings %>% 
  count(minimum_nights) %>% 
  arrange(desc(n)) #arrange in descending order of count to find most common values

The top 3 most common minimum days of stay required by the host are 2 nights with 7,436 observations, 1 night with 6,894 observations and 3 nights with 3,527 observations. Number 4 is 7 nights minimum stay with 1,614 observations.

This clearly shows that hosts are either fine with their guests doing a short-stay around 1-3 days or want them to stay at least a week at 7 days minimum stay.

Ranking place number 7, 8 and 9 are 10, 14 and 20 days respectively. With all this information, it is easy to see that hosts are either okay with short-term stays with frequently new guests and also frequently cleaning and tidying the space, whereas other hosts only want guests for long-term stays (e.g. 1-2 weeks). This is most probably the reason because only long-term stays are profitable for these hosts or they don’t want to tidy the space every second.

To analyse the data, we filter the data set so that it only includes observations with minimum_nights <= 4

#filter minimum nights <= 4
listings <- listings %>% 
  filter(minimum_nights <= 4)

#check
listings %>% 
  count(minimum_nights) %>% 
  arrange(desc(n)) 

NEIGHBOURHOODS

We also find that there are several different neighbourhood variables, namely:

• neigborhood_overview: gives a description
• neighbourhood: all NAs
• neighbourhood_cleansed: indicates which ward listing is in
• neighbourhood_group_cleansed: all NAs

We will eliminate the the two variables that are just NAs and will try to summarize the neighbourhood_cleansed variable into only a few city areas instead of so many wards. Since Cape Town has 116 wards, we decide to split them into 24 sub councils, which deal with local functions for between three and six wards.

listings <- listings %>% 
#creating a new variable to take down sub council names
  mutate(sub_council = case_when(
    neighbourhood_cleansed %in% c("Ward 23","Ward 29", "Ward 32","Ward 104") ~ 1,
    neighbourhood_cleansed %in% c("Ward 6","Ward 7", "Ward 8","Ward 101", "Ward 102", "Ward 111") ~ 2,
    neighbourhood_cleansed %in% c("Ward 1","Ward 4", "Ward 5","Ward 70", "Ward 107", "Ward 113") ~ 3,
    neighbourhood_cleansed %in% c("Ward 25","Ward 26", "Ward 27","Ward 28", "Ward 30") ~ 4,
    neighbourhood_cleansed %in% c("Ward 13","Ward 20", "Ward 24","Ward 31", "Ward 50", "Ward 106") ~ 5,
    neighbourhood_cleansed %in% c("Ward 2","Ward 3", "Ward 9","Ward 10", "Ward 12", "Ward 22") ~ 6,
    neighbourhood_cleansed %in% c("Ward 21","Ward 103", "Ward 105","Ward 112") ~ 7,
    neighbourhood_cleansed %in% c("Ward 83","Ward 85", "Ward 86","Ward 100") ~ 8,
    neighbourhood_cleansed %in% c("Ward 18","Ward 87", "Ward 89","Ward 90", "Ward 91", "Ward 116") ~ 9,
    neighbourhood_cleansed %in% c("Ward 92","Ward 93", "Ward 94","Ward 97", "Ward 98", "Ward 99") ~ 10,
    neighbourhood_cleansed %in% c("Ward 40","Ward 44", "Ward 46","Ward 47") ~ 11,
    neighbourhood_cleansed %in% c("Ward 78","Ward 79", "Ward 81","Ward 82") ~ 12,
    neighbourhood_cleansed %in% c("Ward 34","Ward 35", "Ward 36","Ward 80", "Ward 88") ~ 13,
    neighbourhood_cleansed %in% c("Ward 37","Ward 38", "Ward 39","Ward 41", "Ward 42", "Ward 45") ~ 14,
    neighbourhood_cleansed %in% c("Ward 51","Ward 52", "Ward 53","Ward 55", "Ward 56") ~ 15,
    neighbourhood_cleansed %in% c("Ward 54","Ward 57", "Ward 74","Ward 77", "Ward 115") ~ 16,
    neighbourhood_cleansed %in% c("Ward 48","Ward 49", "Ward 60") ~ 17,
    neighbourhood_cleansed %in% c("Ward 63","Ward 65", "Ward 66","Ward 67", "Ward 68", "Ward 110") ~ 18,
    neighbourhood_cleansed %in% c("Ward 61","Ward 64", "Ward 69") ~ 19,
    neighbourhood_cleansed %in% c("Ward 58","Ward 59", "Ward 62","Ward 71", "Ward 72", "Ward 73") ~ 20,
    neighbourhood_cleansed %in% c("Ward 11","Ward 19", "Ward 108") ~ 21,
    neighbourhood_cleansed %in% c("Ward 14","Ward 16", "Ward 17","Ward 114") ~ 22,
    neighbourhood_cleansed %in% c("Ward 33","Ward 43", "Ward 75","Ward 76") ~ 23,
    TRUE ~ 24
  )) 

After consulting Cape Town experts in our group, we found that the 24 sub councils can be further categorised into 6 neighbourhood areas, which namely are Atlantic seaboard, CBD, Outer suburbs, Southern Suburbs, Cape Flats and Sommerset.

Atlantic seaboard- this term is often used by locals to describe the residential/ commercial areas facing the Atlantic seaboard and in front of table mountain, region is often characterized by luxury apartments and homes and offers many tourist attractions such a views from table mountain and beachfront restaurants

CBD-this region is generally considered the city centre and its surrounding areas characterized as the business/ commercial district of cape town.

Outer suburbs- these are regions generally away from the CBD and seaside, land mainly used for industrial purposes.

Southern Suburbs-Cape Town’s Southern Suburbs lie to the Southeast of the slopes of Table Mountain within rich valleys and vast plains reaching from just south of the Table Bay industrial neighbourhoods in the north to the False Bay coastal suburbs and the Cape Peninsula cliffs to the south, and are crossed North-South by the M3 and M5 freeways. In general, this area is identified as being the more affluent of the Cape Town Metropolis’ sections and includes the city’s most expensive residential neighbourhoods.

Cape Flats- this regions are generally characterized by their townships, this includes Mitchells Plainand and Khayelitsha and guguletho some of the regions largest townships, characterized by being very low income and run down.

Sommerset- Sommerset west is one of the largest coastal towns outside of the CBD, and offers both residential and tourist appeal away from the hussle of the city centre.

#change `sub_council` to character form since it represents a name tag
listings$sub_council <- as.character(listings$sub_council)
#summarize all wards from neighbourhood_cleansed to 6 neighbourhood areas
listings <- listings %>%
  mutate(neighbourhood_simplified = case_when(
    sub_council %in% c(1, 2, 7, 21) ~ "Outer Suburbs", 
    sub_council %in% c(3, 4, 5, 6) ~ "CBD", 
    sub_council %in% c(8, 22) ~ "Sommerset", 
    sub_council %in% c(9, 10, 12, 24) ~ "Cape Flats", 
    sub_council %in% c(11, 13, 14, 17, 18, 23) ~ "Southern Suburbs", 
    TRUE ~ "Atlantic Seaboard"
  ))

#check neighbourhood_simplified compared to sub_council
listings %>%
  count(neighbourhood_simplified, sub_council) %>%
  arrange(desc(n))   

Further, we find these variables regarding location:

• 441 unique values for street
• 259 unique values for city
• 56 unique values for state
• 4 unique values for market
• 2 unique values for country_code
• 2 unique values for country

As several values for market, country_code and country seem odd, we inspect those values further in the dataset.

We find that market includes the following values:

• “Cape Town”
• “Other International”
• “Krabi”
• “D.C.”

However, they all have “Cape Town” as values for the smart location which is why we can either disregard this column or we assume the names are just wrong so that we can change the three odd names to “Cape Town” also.

#change "Other Internatinal", "Krabi", "D.C." to "Cape Town"
listings <- listings %>% 
    mutate(market = case_when(
    market == "Cape Town" ~ "Cape Town",
    TRUE ~ "Cape Town"
  ))

#check
listings %>% 
  count(market) 

When further investigating country_code and country, we find that all observations have the value “ZA” or “South Africa” but one observation which has the values “CG” and “Congo” respectively. As this observation’s smart_location is also “Congo”, we eliminate this observation as a wrongly assigned value.

#filter the observations to ensure they contain "ZA" for 'country_code'
listings <- listings %>% 
    filter(country_code != "CG")

#check
listings %>% 
  count(country_code) 

We find that 6 values within state are called “.Western Cape” instead of “Western Cape” which we will change in the data cleansing as well. @SR

#change ".Western Cape" to "Western Cape"
listings$state[listings$state == ".Western Cape"] <- "Western Cape"
listings$state[listings$state == "Cape town"] <- "Cape Town"
listings$state[listings$state == "Cape Town - Western Cape"] <- "Cape Town"
listings$state[listings$state == "Cape Town Western Cape"] <- "Cape Town"
#check
listings %>% 
  count(state) 

We further find that host_location has several unique values including - “Cape Town, Western Cape, South Africa” - “ZA” - “Parow, Western Cape, South Africa”

listings %>% 
  count(host_location) %>% 
  arrange(desc(n))  

CATEGORICAL VARIABLES

Lastly, we find several categorical or factor variables (numeric or character variables with variables that have a fixed and known set of possible values).

• host_response_time (as specified by Airbnb, e.g. “within an hour”, “within a day”)

• host_response_rate (between 0% and 100%)

• host_acceptance_rate (between 0% and 100%)

• host_is_superhost (TRUE or FALSE)

• host_has_profile_pic (TRUE or FALSE)

• host_identity_verified (TRUE or FALSE)

• neighbourhood_cleansed (limited number of Wards)

• city (limited set of cities that belong to the area)

• state (only one state, “Western Cape”)

• zipcode (limited set of zip codes for the area)

• market (should all be “Cape Town”)

• country_code (should all be “ZA”)

• country (should all be “South Africa”)

TEXT VARIABLES

We further find these following variables which are filled with descriptive text or are just URL-addresses and will, thus, also not be relevant to our analysis.

• name
• summary
• space
• description
• neighborhood_overview
• notes
• transit
• access
• interaction
• house_rules
• host_about
• listing_url
• thumbnail_url (all NAs)
• medium_url (all NAs)
• picture_url
• xl_picture_url (all NAs)
• host_url
• host_thumbnail_url
• host_picture_url

listings_version2 <- listings %>% 
#deselecting all text variables
    dplyr::select(-name, -summary, -space, -description,
                  -neighborhood_overview, -notes, -transit, -access,
                  -interaction, -house_rules, -host_about, -listing_url,
                  -thumbnail_url, -medium_url, -picture_url, -xl_picture_url, 
                  -host_url, -host_thumbnail_url, -host_picture_url)

MISSING VALUES 2

While some variables have 0 missing values, a lot of variables have some missing values. However, we do find 8 variables that miss all values, namely that only (or for one mainly) include NAs (Not Available). We will eliminate these when cleaning the data set.

• host_neighbourhood
• thumbnail_url
• medium_url
• xl_picture_url
• neighbourhood
• neighbourhood_group_cleansed
• jurisdiction_names
• license (mainly NAs except for a few irrelevant values)

listings_version3 <- listings_version2 %>% 
#deselecting variables with only or mainly NAs
    dplyr::select(-host_neighbourhood, -neighbourhood, -neighbourhood_group_cleansed, 
                  -jurisdiction_names, -license)

We can also see that all observations have a value for the experiences_offered variable. As there is only 1 unique value, which is “none”, we will can discard this variable, too as it adds no value to our analysis.

listings_version3 <- listings_version3 %>% 
#deselecting the column `experiences_offered`
  dplyr::select(-experiences_offered)

The target variable is the cost for two people to stay at an Airbnb location for four (4) nights, namely (price+extra_people)*4. So we create a new variable price_4_nights in our dataset ‘listings_version3’.

listings_version3 <- listings_version3 %>% 
#creating a new variable `price_4_nights`
  mutate(price_4_nights = ifelse(guests_included < 2, 4*(price+extra_people) + cleaning_fee, 
                                 price*4+cleaning_fee))

Favstats

Lastly, we will have a look at favstats() of the numeric variables that we specified as relevant earlier:

#favstats related numerical variables to examine rough distribution/outliers
favstats(listings_version3$price)

favstats(listings_version3$price_4_nights)

favstats(listings_version3$number_of_reviews)

favstats(listings_version3$review_scores_rating)

favstats(listings_version3$accommodates)

favstats(listings_version3$longitude)

favstats(listings_version3$latitude)

It seems there exist fairly isolated outliers for these numeric variables.

Meanwhile, we can double check from favstate results that all of the 7 variables have 18954 counts, namely no missing values after we have selected the observations with minimum_nights <= 4.

The maximum of price is 300,009, almost 300 (300,009/988) times higher than the median. The standard deviation for price is 7,537.011, signaling high variability.

For price_4_nights , the max is 1,200,036; the mean (8801) is larger than median (4212), indicating that the distribution could be right-skewed.

For number_of_reviews, mean is similar to Q3 and the maximum is way higher than median. This is possibly because a large number of “0”s exist in the column.

For review_scores_rating, most values are within 94.5 and 99, and the mean is 94, indicating that review_scores_rating are fairly high for Cape Town properties.

For accommodates, most properties can have 2 to 4 people, while the largest accommodates 40.

For longitude(latitude), all values are clustered around “18.4”(“-34.0”). This verifies that we are getting right observations in one certain area, namely Cape Town.

Overall, the results of favstats show there are outliers that could distrupt our analysis. Thus in each visualization, we would like to limit the numerical values to its [quantile 10%, quantile 90%].

Graphs of variables of interest

plot_of_price <- listings_version3 %>% 
  filter(price < quantile(price,0.9), price > quantile(price,0.1)) %>% 
  ggplot(aes(x=price)) + 
  geom_density()+
  labs(title="Distribution of price")+
  scale_x_continuous(breaks = seq(0,4000,500))+
  theme_bw()

plot_of_price

# remove scientific notation
options(scipen=999)

# plot
plot_of_price_4_nights <- listings_version3 %>% 
   filter(cleaning_fee < quantile(price_4_nights,0.9), cleaning_fee > quantile(price_4_nights,0.1)) %>% 
  ggplot(aes(x=price_4_nights)) + 
  geom_density()+
  labs(title="Distribution of price_4_nights")+
  scale_x_continuous(breaks = seq(0,220000,25000))+
  theme_bw()

plot_of_price_4_nights

plot_of_guests_included <- listings_version3 %>% 
   #filter(guests_included < quantile(guests_included,0.9), guests_included > quantile(guests_included,0.1)) %>% 
  ggplot(aes(x=guests_included)) + 
  geom_density()+
  labs(title="Distribution of guest included")+
  scale_x_continuous(breaks = seq(0,20,1))+
  theme_bw()

plot_of_guests_included

plot_of_extra_people <- listings_version3 %>% 
  filter(extra_people < quantile(extra_people,0.9), extra_people > quantile(extra_people,0.1)) %>%
  ggplot(aes(x=extra_people)) + 
  geom_density()+
  labs(title="Distribution of extra people")+
  theme_bw()

plot_of_extra_people

plot_of_minimum_nights <- listings_version3%>% 
  ggplot(aes(x=minimum_nights)) + 
  geom_density()+
  labs(title="Distribution of minimum nights")+
  scale_x_continuous(breaks = seq(0,5,1))+
  theme_bw()

plot_of_minimum_nights

plot_of_number_of_reviews <- listings_version3%>% 
  filter(number_of_reviews < quantile(number_of_reviews,0.9), number_of_reviews > quantile(number_of_reviews,0.1)) %>%
  ggplot(aes(x=number_of_reviews)) + 
  geom_density()+
  labs(title="Distribution of number of reviews")+
  scale_x_continuous(breaks = seq(0,55,10))+
  theme_bw()

plot_of_number_of_reviews 

plot_of_review_scores_rating <- listings_version3%>% 
  filter(review_scores_rating < quantile(review_scores_rating,0.9), review_scores_rating > quantile(review_scores_rating,0.1)) %>%
  ggplot(aes(x=review_scores_rating)) + 
  geom_density()+
  labs(title="Distribution of review scores rating")+
  theme_bw()

plot_of_review_scores_rating

plot_of_host_listings_count <- listings_version3%>% 
  ggplot(aes(x=host_listings_count)) + 
  geom_density()+
  labs(title="Distribution of host listings count")+
  xlim(c(0,50)) +
  theme_bw()

plot_of_host_listings_count

plot_of_security_deposit <- listings_version3%>% 
  ggplot(aes(x=security_deposit)) + 
  geom_density()+
  labs(title="Distribution of security deposit")+
  xlim(c(0,10)) +
  theme_bw()

plot_of_security_deposit

plot_of_accommodates <- listings_version3 %>% 
  ggplot(aes(x=accommodates)) + 
  geom_histogram(binwidth = 1)+
  labs(title="Distribution of accommodates")+
  xlim(0,15)+
  #scale_x_continuous(breaks = seq(1,14,1))+
  theme_bw()

plot_of_accommodates

plot_of_accommodates_by_prop_type <- plot_of_accommodates+
  facet_wrap(~prop_type_simplified,scale="free")+
  labs(title="Distribution of accommodates by property type")

plot_of_accommodates_by_prop_type

listings  %>% 
  filter(prop_type_simplified=="Villa") %>% 
  count()

plot_of_accommodates_villa <- listings_version3 %>% 
  filter(prop_type_simplified=="Villa") %>% 
  ggplot(aes(x=accommodates)) + 
  geom_histogram(binwidth = 1)+
  labs(title="Distribution of villa accommodates")+
  #xlim(0,15)+
  scale_x_continuous(breaks = seq(1,20,1))+
  theme_bw()

plot_of_accommodates_villa

data_of_room_type <- listings_version3%>% 
 group_by(room_type) %>% 
 summarise(count = n()) %>%
 mutate(room_type = fct_reorder(room_type, desc(count)))

plot_of_room_type <-  data_of_room_type %>% 
  ggplot(aes(x=room_type,y=count)) +
  geom_bar(stat = "identity")+
  geom_text(aes(label=count), vjust=1.6, color="white", size=3.5)+
  theme_bw()

plot_of_room_type

data_of_neighbourhood_simplified <- listings_version3 %>% 
 group_by(neighbourhood_simplified) %>% 
 summarise(count = n()) %>%
 mutate(neighbourhood_simplified = fct_reorder(neighbourhood_simplified, desc(count)))

plot_of_neighbourhood_simplified <- data_of_neighbourhood_simplified %>% 
  ggplot(aes(x=neighbourhood_simplified,y=count)) +
  labs(title="Most properties are located in Atlantic Seaboard")+
  geom_bar(stat = "identity")+
  theme_bw()


plot_of_neighbourhood_simplified

data_of_prop_type_simplified <- listings_version3 %>% 
 group_by(prop_type_simplified) %>% 
 summarise(count = n()) %>%
 mutate(prop_type_simplified= fct_reorder(prop_type_simplified,desc(count)))

plot_of_prop_type_simplified <- data_of_prop_type_simplified %>% 
  ggplot(aes(x=prop_type_simplified,y=count)) +
  labs(title="Most properties are apartment, followed by house")+
  geom_bar(stat = "identity")+
  theme_bw()

plot_of_prop_type_simplified

plot_of_host_is_superhost <- listings_version3%>% 
  filter(!is.na(host_is_superhost)) %>% 
  ggplot(aes(x=host_is_superhost)) +
  labs(title="Most hosts are not superhost")+
  geom_bar()+
  theme_bw()

plot_of_host_is_superhost

plot_of_host_identity_verified <- listings_version3%>% 
  filter(!is.na(host_identity_verified)) %>% 
  ggplot(aes(x=host_identity_verified)) +
  labs(title="Most hosts are not verified")+
  geom_bar()+
  theme_bw()

plot_of_host_identity_verified

# check out the type of cancellation policy
typeof(listings_version3$cancellation_policy)

## [1] "character"

# make it a factor variable and prepare the data for plotting
data_of_cancellation_policy <- listings_version3 %>% 
 dplyr::select(cancellation_policy)  %>% 
 mutate(cancellation_policy_f = as.factor(cancellation_policy)) %>% 
 group_by(cancellation_policy_f) %>% 
 summarise(count = n()) %>%
 mutate(cancellation_policy_f= fct_reorder(cancellation_policy_f,desc(count)))

# plot 
plot_of_cancellation_policy <- data_of_cancellation_policy %>% 
  ggplot(aes(x=cancellation_policy_f,y=count)) +
  geom_bar(stat = "identity")+
  theme_bw()

plot_of_cancellation_policy 

Correlation matrix

To measure correlation between variables, we use correlation matrix to generate realationship plots and correlation coefficents, which measure both the strength and direction of the linear relationship between two continuous variables.

#quantile(listings_version3$price, probs = c(0.1,0.9))
#quantile(listings_version3$cleaning_fee, probs = c(0.1,0.9))
#quantile(listings_version3$extra_people, probs = c(0.1,0.9))
#quantile(listings_version3$number_of_reviews, probs = c(0.1,0.9))
#quantile(listings_version3$review_scores_rating, probs = c(0.1,0.9))
listings_version3_filtered <- listings_version3 %>% 
                    filter(price_4_nights < quantile(price_4_nights,0.9), price_4_nights > quantile(price_4_nights,0.1),
                           number_of_reviews < quantile(number_of_reviews,0.9), number_of_reviews > quantile(number_of_reviews,0.1),
                          review_scores_rating < quantile(review_scores_rating,0.9), review_scores_rating > quantile(review_scores_rating,0.1),
                          accommodates < quantile(accommodates,0.9), accommodates > quantile(accommodates,0.1))

# produce scatterplot-correlation matrix using GGally::ggpairs()
# remove scientific notation
options(scipen=999)
# listings
correlation_matrix1 <- listings_version3_filtered %>% 
  dplyr::select(price_4_nights, number_of_reviews, review_scores_rating, accommodates) %>% 
  GGally::ggpairs(alpha = 0.4)
   
#load library here
library(here)
ggsave("correlation_matrix1.jpg",plot = correlation_matrix1 ,path=here::here("images"), width = 17, height = 10)  
  knitr::include_graphics(here::here("images", "correlation_matrix1.jpg"))

# produce scatterplot-correlation matrix using GGally::ggpairs()
# remove scientific notation
options(scipen=999)
correlation_matrix2 <-listings_version3_filtered %>% 
  dplyr::select(price_4_nights, number_of_reviews, review_scores_rating, accommodates,neighbourhood_simplified) %>% 
  GGally::ggpairs(aes(color = neighbourhood_simplified), alpha = 0.4)


#load library here
library(here)
ggsave("correlation_matrix2.jpg", plot = correlation_matrix2, path=here::here("images"), width = 17, height = 10)  
  knitr::include_graphics(here::here("images", "correlation_matrix2.jpg"))

# produce scatterplot-correlation matrix using GGally::ggpairs()
# remove scientific notation
options(scipen=999)
correlation_matrix3 <-listings_version3_filtered %>% 
  dplyr::select(price_4_nights, number_of_reviews, review_scores_rating, accommodates,prop_type_simplified) %>% 
  GGally::ggpairs(aes(color =  prop_type_simplified), alpha = 0.4)

# load library here
library(here)
ggsave("correlation_matrix3.jpg", plot = correlation_matrix3, path=here::here("images"), width = 17, height = 10)  
  knitr::include_graphics(here::here("images", "correlation_matrix3.jpg"))

# produce scatterplot-correlation matrix using GGally::ggpairs()
# remove scientific notation
options(scipen=999)
correlation_matrix4 <-listings_version3_filtered %>% 
# filter variables of interest 
  dplyr::select(price_4_nights, number_of_reviews, review_scores_rating, accommodates,room_type) %>% 
  GGally::ggpairs(aes(color = as.factor(room_type)), alpha = 0.4)

# load library here
library(here)
ggsave("correlation_matrix4.jpg", plot = correlation_matrix4, path=here::here("images"), width = 17, height = 10)  
knitr::include_graphics(here::here("images", "correlation_matrix4.jpg"))

Correlations conditional on categorical variable

price_4_nights_number_of_reviews_1 <- listings_version3 %>% 
  filter(price_4_nights < quantile(price_4_nights,0.9), price_4_nights > quantile(price_4_nights,0.1)) %>% 
  filter(number_of_reviews < quantile(number_of_reviews,0.9), number_of_reviews > quantile(number_of_reviews,0.1)) %>%
  ggplot2::ggplot(aes(x = number_of_reviews , y = price_4_nights))+
  geom_point()+
  geom_smooth()+
  facet_wrap(~neighbourhood_simplified)+
  labs(title = "'CBD' and 'Atlantic Seaboard' sees strong relationship between price_4_nights and number_of_reviews", 
       subtitle = "correlation comparison conditional on neighbourhoods",
       x = "number_of_reviews")+
  theme_bw()+
  NULL


ggsave("price_4_nights_number_of_reviews_1.jpg", plot = price_4_nights_number_of_reviews_1, path=here::here("images"), width = 15, height = 10)  
knitr::include_graphics(here::here("images", "price_4_nights_number_of_reviews_1.jpg"))

price_4_nights_accommodates_byneighborhood <- listings_version3 %>% 
  filter(price_4_nights < quantile(price_4_nights,0.9), price_4_nights > quantile(price_4_nights,0.1)) %>% 
  filter(accommodates < quantile(accommodates,0.9), accommodates > quantile(accommodates,0.1)) %>% 
  ggplot2::ggplot(aes(x = as.factor(accommodates), y = price_4_nights))+
  geom_boxplot()+
  facet_wrap(~neighbourhood_simplified)+
  #scale_x_continuous(limits = c(75,300))+
  #scale_y_continuous(limits = c(0,3000))+
  labs(title = " 'Atlantic Seaboard' sees highest average price_4_nights across accommodates", 
       subtitle = "Correlation comparison conditional on neighbourhoods",
       x = "accommodates")+
  theme_bw()+
  NULL

ggsave("price_4_nights_accommodates_byneighborhood.jpg", plot = price_4_nights_accommodates_byneighborhood, path=here::here("images"), width = 15, height = 10)  
knitr::include_graphics(here::here("images", "price_4_nights_accommodates_byneighborhood.jpg"))

price_4_nights_accommodates_byprop<- listings_version3 %>% 
  filter(price_4_nights < quantile(price_4_nights,0.9), price_4_nights > quantile(price_4_nights,0.1)) %>% 
  filter(accommodates < quantile(accommodates,0.9), accommodates > quantile(accommodates,0.1)) %>% 
  ggplot2::ggplot(aes(x = as.factor(accommodates), y = price_4_nights))+
  geom_boxplot()+
  facet_wrap(~prop_type_simplified)+
  #scale_x_continuous(limits = c(75,300))+
  #scale_y_continuous(limits = c(0,3000))+
  labs(title = " 'House' sees most significant positive relationship ", 
       subtitle = "Correlation comparison conditional on property type",
       x = "accommodates")+
  theme_bw()+
  NULL

ggsave("price_4_nights_accommodates_byprop.jpg", plot = price_4_nights_accommodates_byprop, path=here::here("images"), width = 15, height = 10)  
knitr::include_graphics(here::here("images", "price_4_nights_accommodates_byprop.jpg"))

The basic map

The variables we use are price, longitude, latitude, listing_url, and property_type. The label shows the prop_type_simplified ; the pop up shows its price per night and the corresponding url.

# load the leaflet package
library(leaflet)
# just use price per se
# add layers indicating the property type
library(RColorBrewer)
# assign color palette
# we can do colorNumeric or colorQuantile but to display price, we choose the former
pal <- colorNumeric(
  palette = "PuBuGn",
  domain = c(0,3500),
  n = 5
  )

# create a  basic map
map1 <- leaflet(data =listings_formapping_final
        ) %>% 
  addProviderTiles("OpenStreetMap.Mapnik") %>% 
  # set the initial view
  setView( lat=-34, lng=18.5 , zoom=8) %>%
  # add the circles
  addCircleMarkers(lng = ~longitude, 
                   lat = ~latitude, 
                   radius = 1, 
                   color = ~pal(price),
                   fillOpacity = 0.4, 
                   # popup shows price per night and url
                   popup = ~paste0("<b>","$",price," per night","</b>"," <br/>",listing_url),
                   label = ~prop_type_simplified) %>% 
  # add legends
    addLegend(pal=pal, values=~price, opacity=0.9, 
              title = "Price per night", position = "bottomleft")

Airbnb listings in Cape Town

The map with layers of property types

For a clearer view, we use Leaflet’s layers control feature for this new mapping. We can toggle the visibility of different property types to show by clicking on the layer control. And the properties are colored based on its price.

# check the property type simplified (we have 5 groups)
listings_formapping_final %>% 
  count(prop_type_simplified) %>% 
  arrange(desc(n))  

# create a more complicated map with layers indicating property types
map2_base <- leaflet(data =listings_formapping_final
        ) %>% 
  addProviderTiles("OpenStreetMap.Mapnik") %>% 
  # set the initial view
  setView( lat=-34, lng=18.5 , zoom=10) 
  

# create individual datasets for each property type
m2_apartment <- listings_formapping_final %>% filter(prop_type_simplified == "Apartment")
m2_other <- listings_formapping_final %>% filter(prop_type_simplified == "Other")
m2_house <- listings_formapping_final %>% filter(prop_type_simplified == "House")
m2_guest <- listings_formapping_final %>% filter(prop_type_simplified == "Guest suite")
m2_villa <- listings_formapping_final %>% filter(prop_type_simplified == "Villa")


  # add the circles for Apartment
map2_with_layers <- map2_base %>% addCircleMarkers(data = m2_apartment,
                   lng = ~longitude, 
                   lat = ~latitude, 
                   radius = 1, 
                   color = ~pal(price),
                   fillOpacity = 0.4, 
                   # set the pop up to be listing url
                   # popup shows price per night and url
                   popup = ~paste0("<b>","$",price," per night","</b>"," <br/>",listing_url),
                   # set the label to be property_type
                   label = ~property_type,
                   group = "Apartment") %>% 
  
  # add the circles for House
    addCircleMarkers(data = m2_house,
                   lng = ~longitude, 
                   lat = ~latitude, 
                   radius = 1, 
                   color = ~pal(price),
                   fillOpacity = 0.4, 
                  # popup shows price per night and url
                   popup = ~paste0("<b>","$",price," per night","</b>"," <br/>",listing_url),
                   label = ~property_type,
                   group = "House") %>% 
  
  # add the circles for Guest suite
    addCircleMarkers(data = m2_guest,
                   lng = ~longitude, 
                   lat = ~latitude, 
                   radius = 1, 
                   color = ~pal(price),
                   fillOpacity = 0.4, 
                   # popup shows price per night and url
                   popup = ~paste0("<b>","$",price," per night","</b>"," <br/>",listing_url),
                   label = ~property_type,
                   group = "Guest suite") %>% 
  
  # add the circles for Villa
    addCircleMarkers(data = m2_villa,
                   lng = ~longitude, 
                   lat = ~latitude, 
                   radius = 1, 
                   color = ~pal(price),
                   fillOpacity = 0.4, 
                   # popup shows price per night and url
                   popup = ~paste0("<b>","$",price," per night","</b>"," <br/>",listing_url),
                   label = ~property_type,
                   group = "Villa") %>% 
  
    # add the circles for Other
    addCircleMarkers( data = m2_other,
                lng = ~longitude, 
                   lat = ~latitude, 
                   radius = 1, 
                   color = ~pal(price),
                   fillOpacity = 0.4, 
                  # popup shows price per night and url
                   popup = ~paste0("<b>","$",price," per night","</b>"," <br/>",listing_url),
                # here we used the original property type for group other
                   label = ~property_type,
                   group = "Other") %>% 
  
  # add layer control
  addLayersControl(overlayGroups = c("Apartment","House","Guest suite","Villa","Other")) %>% 
  # add legends
    addLegend(pal=pal, values=~price, opacity=0.9, 
              title = "Price per night", position = "bottomleft")

Airbnb listings in Cape Town by property type

Initial Findings

Based on the map above, we observe location and price difference between various types of properties.

• For apartments, those near the coastline, for example Three Anchor Bay (belongs to Atlantic seaboard), have higher price per night.
• For houses, those located in Newlands, Bishopscoutr (belong to Southern Suburbs) are more expensive.
• For guest suites, those located in the Glen, Bakoven, and Rontree (also belong to Atlantic seaboard) cost more.
• For villas, the properties seem rather scattered around and are mostly priced above $1,500 per night.
• For properties that belong to other types, the distribution looks similar to that of apartments.

To conclude:

• In terms of location, properties near the coastlines (Atlantic seaboard) are more likely to be expensive because they have better views.
• In terms of the property type and price, houses and villas generally have higher price per night, while guest suites have lower price per night.

Un-transformed model

model1 <-lm(price_4_nights~prop_type_simplified+number_of_reviews+review_scores_rating, data=listings_model)

summary(model1)

## 
## Call:
## lm(formula = price_4_nights ~ prop_type_simplified + number_of_reviews + 
##     review_scores_rating, data = listings_model)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
##  -54873   -4150   -2312     346 1192468 
## 
## Coefficients:
##                                 Estimate Std. Error t value
## (Intercept)                      2658.44    2594.60    1.02
## prop_type_simplifiedGuest suite -2491.34     830.46   -3.00
## prop_type_simplifiedHouse        1658.67     526.61    3.15
## prop_type_simplifiedOther         633.15     531.96    1.19
## prop_type_simplifiedVilla       49620.25    1123.39   44.17
## number_of_reviews                 -33.38       6.63   -5.03
## review_scores_rating               45.21      27.48    1.65
##                                             Pr(>|t|)    
## (Intercept)                                   0.3056    
## prop_type_simplifiedGuest suite               0.0027 ** 
## prop_type_simplifiedHouse                     0.0016 ** 
## prop_type_simplifiedOther                     0.2340    
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews                         0.00000049 ***
## review_scores_rating                          0.0999 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 25200 on 15157 degrees of freedom
## Multiple R-squared:  0.121,  Adjusted R-squared:  0.121 
## F-statistic:  348 on 6 and 15157 DF,  p-value: <0.0000000000000002

plot(model1)

hist(rstandard(model1))

vif(model1)

##                      GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified 1.02  4            1.00
## number_of_reviews    1.03  1            1.01
## review_scores_rating 1.02  1            1.01

bptest(model1)

## 
##  studentized Breusch-Pagan test
## 
## data:  model1
## BP = 232, df = 6, p-value <0.0000000000000002

outlierTest(model1)

##       rstudent
## 11271     51.3
## 7629      27.0
## 14925     25.7
## 5788      24.6
## 10099     22.8
## 11630     22.4
## 7630      21.9
## 7628      21.9
## 7448      21.9
## 10829     21.9
##                                                                                                                                                     unadjusted p-value
## 11271 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
## 7629  0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000126
## 14925 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000007983499999999998
## 5788  0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000002248499999999999268915646284
## 10099 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000008514699999999998887407205483270459632269445262
## 11630 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000017653999999999997139580909755029453942195577748392
## 7630  0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000583649999999999923186102589075707416687641975890849565
## 7628  0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000600439999999999847585238188017319190671962500168875057
## 7448  0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000606859999999999865730038734441742969341264879935444477
## 10829 0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000606859999999999865730038734441742969341264879935444477
##                                                                                                                                                       Bonferroni p
## 11271 0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
## 7629  0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000191
## 14925 0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000012105999999999995
## 5788  0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000003409699999999998875200300255
## 10099 0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000012911999999999998245735330732217302080912452105
## 11630 0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000026770999999999996184072387504995849260463826623226
## 7630  0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000885039999999999876096780013780954653828676189386586889
## 7628  0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000910509999999999815553244863287992384541169941610996367
## 7448  0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000920249999999999885916981885468414196076770051294187062
## 10829 0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000920249999999999885916981885468414196076770051294187062

Initial analysis of model1 of untransformed price_4_nights against some intuitive explanatory variables: Property Type(simplified),Number of Reviews and Review Score Rating yields a rather weak model with serious violations of underlying assumptions.

The Adjusted R squared reveals that this model only explains approximately 12% of the variability in the dependent variable. It assumes Apartment property types as a base and from the summary we can see that the property being a guest suite is a significant variable and reduces the price by 2491.337, while renting a house or villa is also significant factor and increases the price in comparison to renting an apartment by 1658.674 and 49620.250 respectively on average over 4 nights for 2 people. Other property types are not significant. Number of reviews also proved to be significant ,however, the review rating does not seem to be significant at the 5% level. It’s interesting to note that more reviews equates to a lower price (-33.38 per review)whereas the quality of reviews does not seem to have a significant effect.

Plotting the standardized residuals and Normal-QQ plot reveals the non-normality of the data, we then conducted a Breusch-Pagan test (BP test) which confirmed suspicion of Heteroscedasticity, This can be seen in the residuals vs fitted plot. These violations of underlying assumptions of multiple-regression make this model flawed and thus we will no longer consider the untransformed model.It can be noted that no particular observations was beyond 0.5 Cook’s Distance however, when running an outlier test, this revealed 10 possible outliers that we may want to consider removing to correct the underlying issues. However, as there are 15164 observations we deemed that removing these 10 outliers may have little to no effect.

Log transformed model

model1t <-lm(log(price_4_nights)~prop_type_simplified+number_of_reviews+review_scores_rating,data=listings_model) #running the same variables on a log transformed test 

summary(model1t)

## 
## Call:
## lm(formula = log(price_4_nights) ~ prop_type_simplified + number_of_reviews + 
##     review_scores_rating, data = listings_model)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -2.651 -0.469 -0.078  0.400  5.563 
## 
## Coefficients:
##                                  Estimate Std. Error t value
## (Intercept)                      7.839536   0.079678   98.39
## prop_type_simplifiedGuest suite -0.425330   0.025503  -16.68
## prop_type_simplifiedHouse        0.097761   0.016172    6.05
## prop_type_simplifiedOther       -0.009298   0.016336   -0.57
## prop_type_simplifiedVilla        1.634476   0.034498   47.38
## number_of_reviews               -0.001187   0.000204   -5.83
## review_scores_rating             0.006391   0.000844    7.57
##                                             Pr(>|t|)    
## (Intercept)                     < 0.0000000000000002 ***
## prop_type_simplifiedGuest suite < 0.0000000000000002 ***
## prop_type_simplifiedHouse          0.000000001526682 ***
## prop_type_simplifiedOther                       0.57    
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews                  0.000000005775807 ***
## review_scores_rating               0.000000000000038 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.774 on 15157 degrees of freedom
## Multiple R-squared:  0.16,   Adjusted R-squared:  0.16 
## F-statistic:  481 on 6 and 15157 DF,  p-value: <0.0000000000000002

plot(model1t)

hist(rstandard(model1t))

vif(model1t)

##                      GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified 1.02  4            1.00
## number_of_reviews    1.03  1            1.01
## review_scores_rating 1.02  1            1.01

bptest(model1t)

## 
##  studentized Breusch-Pagan test
## 
## data:  model1t
## BP = 767, df = 6, p-value <0.0000000000000002

Due to the non-normality issues of the underlying residuals we analyze the log transformation to the dependent variable. This seems to correct the non-normality of the residuals as shown by the increase in slope and fit of the Normal-QQ plot, and more normal looking distribution of the standard residuals histogram. We also have a higher Adjusted R squared(approx 16%) and thus conclude that the log transformed model is a better estimate of price for 4 nights. We will thus continue the analysis considering the log transformed model. However we note that the residuals vs fitted plot again reveals heteroscedasticity issues that should be corrected. The VIF test reveals no multi-collinearity issues and we once again run an outlier test to identify possible observations that could be removed in order to improve the model.

Box-Cox Model

model1_box <-lm(bcPower(price_4_nights,lambda=-0.25)~prop_type_simplified+number_of_reviews+review_scores_rating, data=listings_model) #box cox lm model on 3 variables

summary(model1_box)

## 
## Call:
## lm(formula = bcPower(price_4_nights, lambda = -0.25) ~ prop_type_simplified + 
##     number_of_reviews + review_scores_rating, data = listings_model)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.3670 -0.0518 -0.0020  0.0542  0.3729 
## 
## Coefficients:
##                                   Estimate Std. Error t value
## (Intercept)                      3.4358547  0.0090650  379.02
## prop_type_simplifiedGuest suite -0.0525631  0.0029015  -18.12
## prop_type_simplifiedHouse        0.0058936  0.0018399    3.20
## prop_type_simplifiedOther       -0.0019794  0.0018586   -1.07
## prop_type_simplifiedVilla        0.1536879  0.0039249   39.16
## number_of_reviews               -0.0001121  0.0000232   -4.84
## review_scores_rating             0.0007657  0.0000960    7.98
##                                             Pr(>|t|)    
## (Intercept)                     < 0.0000000000000002 ***
## prop_type_simplifiedGuest suite < 0.0000000000000002 ***
## prop_type_simplifiedHouse                     0.0014 ** 
## prop_type_simplifiedOther                     0.2869    
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews                 0.0000013272095799 ***
## review_scores_rating              0.0000000000000016 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.088 on 15157 degrees of freedom
## Multiple R-squared:  0.125,  Adjusted R-squared:  0.124 
## F-statistic:  360 on 6 and 15157 DF,  p-value: <0.0000000000000002

plot(model1_box)

hist(rstandard(model1_box))

bptest(model1_box)

## 
##  studentized Breusch-Pagan test
## 
## data:  model1_box
## BP = 685, df = 6, p-value <0.0000000000000002

vif(model1_box)

##                      GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified 1.02  4            1.00
## number_of_reviews    1.03  1            1.01
## review_scores_rating 1.02  1            1.01

outlierTest(model1_box)

## No Studentized residuals with Bonferroni p < 0.05
## Largest |rstudent|:
##       rstudent unadjusted p-value Bonferroni p
## 11271     4.24          0.0000226        0.343

While the box-cox transformation seemed to correct the underlying non-normality the most succesfully (see QQ-plot and histogram of residuals ),this transformation yielded a rather weak Adjusted R-squared of 12.43% meaning very little variability is being captured in this model. The log transformation seems to yield a stronger result while still maintaining normality. This model also seems to reflect heteroscedasticity issues(shown by residuals vs fitted plot and failing the BP test), however we will continue this analysis using both the box-cox and log transformed model.VIF test reveals no multi-colinearity issues and we once again run an outlier test to identify possible observations that could be removed in order to improve the model.

Log transformation model with room type

model2_log <-lm(log(price_4_nights)~prop_type_simplified+number_of_reviews+review_scores_rating+room_type, data=listings_model)

summary(model2_log)

## 
## Call:
## lm(formula = log(price_4_nights) ~ prop_type_simplified + number_of_reviews + 
##     review_scores_rating + room_type, data = listings_model)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -2.781 -0.460 -0.093  0.372  5.968 
## 
## Coefficients:
##                                  Estimate Std. Error t value
## (Intercept)                      8.098284   0.073837  109.68
## prop_type_simplifiedGuest suite -0.281179   0.023745  -11.84
## prop_type_simplifiedHouse        0.310308   0.015535   19.98
## prop_type_simplifiedOther        0.183902   0.016143   11.39
## prop_type_simplifiedVilla        1.662733   0.031882   52.15
## number_of_reviews               -0.002412   0.000190  -12.69
## review_scores_rating             0.004737   0.000781    6.07
## room_typeHotel room             -0.056496   0.042113   -1.34
## room_typePrivate room           -0.700470   0.014317  -48.93
## room_typeShared room            -1.164642   0.069776  -16.69
##                                             Pr(>|t|)    
## (Intercept)                     < 0.0000000000000002 ***
## prop_type_simplifiedGuest suite < 0.0000000000000002 ***
## prop_type_simplifiedHouse       < 0.0000000000000002 ***
## prop_type_simplifiedOther       < 0.0000000000000002 ***
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews               < 0.0000000000000002 ***
## review_scores_rating                    0.0000000013 ***
## room_typeHotel room                             0.18    
## room_typePrivate room           < 0.0000000000000002 ***
## room_typeShared room            < 0.0000000000000002 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.715 on 15154 degrees of freedom
## Multiple R-squared:  0.283,  Adjusted R-squared:  0.282 
## F-statistic:  664 on 9 and 15154 DF,  p-value: <0.0000000000000002

plot(model2_log)

hist(rstandard(model2_log))

bptest(model2_log)

## 
##  studentized Breusch-Pagan test
## 
## data:  model2_log
## BP = 389, df = 9, p-value <0.0000000000000002

vif(model2_log)

##                      GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified 1.23  4            1.03
## number_of_reviews    1.05  1            1.02
## review_scores_rating 1.02  1            1.01
## room_type            1.24  3            1.04

outlierTest(model2_log)

##       rstudent    unadjusted p-value     Bonferroni p
## 11271     8.37 0.0000000000000000633 0.00000000000096
## 14925     8.18 0.0000000000000003111 0.00000000000472
## 5788      7.43 0.0000000000001144000 0.00000000173480
## 12609     6.69 0.0000000000235000000 0.00000035636000
## 11630     6.32 0.0000000002707100000 0.00000410500000
## 9329      5.50 0.0000000385180000000 0.00058409000000
## 10830     5.47 0.0000000448000000000 0.00067935000000
## 15017     5.33 0.0000001007200000000 0.00152730000000
## 492       5.26 0.0000001465400000000 0.00222210000000
## 4502      5.16 0.0000002488100000000 0.00377300000000

The log transformed model showed significant improvement with the addition of the room type independent variable. Adjusted R-squared significantly increased from 16% to 28%. Using room type Entire Apt and property type apartment as a base all explanatory variables are now significant at the 5 % level except for room_type Hotel. Our intercept marginally increases meaning there was an increase in average price. The addition of room types also alters the slope of the co-efficient for some of the other factors. Property types Guest suite slope reduces in magnitude while all other factors in property type appear to increase in magnitude with property type ‘Other’ becoming significant. Review score rating also becomes significant which it was not before. And while the fitted vs residuals plot also showed improvement we note that it again fails the BP test and we have not solved the heteroscedasticity issues. VIF reveals no multi-colinearity issues and we once again identify possible outliers, which we will continue to do for the remaining models.

Box cox transformation model with room type

model2_box <-lm( bcPower(price_4_nights,lambda=-0.25)~prop_type_simplified+
                   number_of_reviews+
                   review_scores_rating+
                   room_type, data=listings_model) 

summary(model2_box)

## 
## Call:
## lm(formula = bcPower(price_4_nights, lambda = -0.25) ~ prop_type_simplified + 
##     number_of_reviews + review_scores_rating + room_type, data = listings_model)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.3792 -0.0518 -0.0043  0.0497  0.4693 
## 
## Coefficients:
##                                   Estimate Std. Error t value
## (Intercept)                      3.4678496  0.0082809  418.78
## prop_type_simplifiedGuest suite -0.0349367  0.0026630  -13.12
## prop_type_simplifiedHouse        0.0319065  0.0017422   18.31
## prop_type_simplifiedOther        0.0220273  0.0018104   12.17
## prop_type_simplifiedVilla        0.1571410  0.0035756   43.95
## number_of_reviews               -0.0002632  0.0000213  -12.35
## review_scores_rating             0.0005601  0.0000876    6.40
## room_typeHotel room             -0.0099548  0.0047230   -2.11
## room_typePrivate room           -0.0855355  0.0016056  -53.27
## room_typeShared room            -0.1534480  0.0078254  -19.61
##                                             Pr(>|t|)    
## (Intercept)                     < 0.0000000000000002 ***
## prop_type_simplifiedGuest suite < 0.0000000000000002 ***
## prop_type_simplifiedHouse       < 0.0000000000000002 ***
## prop_type_simplifiedOther       < 0.0000000000000002 ***
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews               < 0.0000000000000002 ***
## review_scores_rating                   0.00000000016 ***
## room_typeHotel room                            0.035 *  
## room_typePrivate room           < 0.0000000000000002 ***
## room_typeShared room            < 0.0000000000000002 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.0802 on 15154 degrees of freedom
## Multiple R-squared:  0.274,  Adjusted R-squared:  0.273 
## F-statistic:  635 on 9 and 15154 DF,  p-value: <0.0000000000000002

plot(model2_box)

hist(rstandard(model2_box))

bptest(model2_box)

## 
##  studentized Breusch-Pagan test
## 
## data:  model2_box
## BP = 286, df = 9, p-value <0.0000000000000002

vif(model2_box)

##                      GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified 1.23  4            1.03
## number_of_reviews    1.05  1            1.02
## review_scores_rating 1.02  1            1.01
## room_type            1.24  3            1.04

outlierTest(model2_box)

##       rstudent unadjusted p-value Bonferroni p
## 14925     5.89      0.00000000401    0.0000608
## 11271     5.27      0.00000014067    0.0021332
## 12609     5.20      0.00000019905    0.0030184
## 5788      4.99      0.00000060215    0.0091310
## 9884     -4.73      0.00000221980    0.0336620

The box-cox transformed model also showed significant improvement with the addition of the room type independent variable. Adjusted R-squared significantly increased from ~12.5% to ~27%. Using room type Entire Apt and property type apartment as a base model all explanatory variables are now significant at the 5 % level. Just as in the log model we note that the data still reflects heteroscedasticity issues. Both the log transformed and box-cox, however, seem to have maintained normality in their residuals so we will continue assessing both models by next considering if number of bathrooms, bedrooms, beds, or size of the house (accomodates) significantly predict (price_4_nights)Price for 4 nights. We can conclude our hypothesis was correct and that room type is a significant predictor in Price for 4 nights

Box-Cox model with beds,bathrooms,bedrooms and size

model3_box_BBB <-lm(bcPower(price_4_nights,lambda = -0.25)~prop_type_simplified+ #adding bed bathrooms, bedrooms and size(accoommodates)
                      number_of_reviews+
                      review_scores_rating+
                      room_type+bathrooms+
                      #bedrooms+
                      beds+
                      accommodates, 
                    data=listings_model)


summary(model3_box_BBB)

## 
## Call:
## lm(formula = bcPower(price_4_nights, lambda = -0.25) ~ prop_type_simplified + 
##     number_of_reviews + review_scores_rating + room_type + bathrooms + 
##     beds + accommodates, data = listings_model)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.5040 -0.0421 -0.0027  0.0411  0.4341 
## 
## Coefficients:
##                                   Estimate Std. Error t value
## (Intercept)                      3.3989902  0.0074633  455.43
## prop_type_simplifiedGuest suite -0.0297488  0.0023626  -12.59
## prop_type_simplifiedHouse       -0.0033588  0.0016416   -2.05
## prop_type_simplifiedOther        0.0086542  0.0016166    5.35
## prop_type_simplifiedVilla        0.0569981  0.0035199   16.19
## number_of_reviews               -0.0001963  0.0000189  -10.40
## review_scores_rating             0.0005617  0.0000781    7.20
## room_typeHotel room              0.0086523  0.0041835    2.07
## room_typePrivate room           -0.0490164  0.0015478  -31.67
## room_typeShared room            -0.1158545  0.0070262  -16.49
## bathrooms                        0.0141448  0.0007323   19.32
## beds                            -0.0066829  0.0005019  -13.32
## accommodates                     0.0180558  0.0004363   41.39
##                                             Pr(>|t|)    
## (Intercept)                     < 0.0000000000000002 ***
## prop_type_simplifiedGuest suite < 0.0000000000000002 ***
## prop_type_simplifiedHouse                      0.041 *  
## prop_type_simplifiedOther           0.00000008765063 ***
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews               < 0.0000000000000002 ***
## review_scores_rating                0.00000000000065 ***
## room_typeHotel room                            0.039 *  
## room_typePrivate room           < 0.0000000000000002 ***
## room_typeShared room            < 0.0000000000000002 ***
## bathrooms                       < 0.0000000000000002 ***
## beds                            < 0.0000000000000002 ***
## accommodates                    < 0.0000000000000002 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.0708 on 15069 degrees of freedom
##   (82 observations deleted due to missingness)
## Multiple R-squared:  0.432,  Adjusted R-squared:  0.432 
## F-statistic:  955 on 12 and 15069 DF,  p-value: <0.0000000000000002

plot(model3_box_BBB)

hist(rstandard(model3_box_BBB))

bptest(model3_box_BBB)

## 
##  studentized Breusch-Pagan test
## 
## data:  model3_box_BBB
## BP = 1063, df = 12, p-value <0.0000000000000002

vif(model3_box_BBB)

##                      GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified 1.67  4            1.07
## number_of_reviews    1.05  1            1.02
## review_scores_rating 1.02  1            1.01
## room_type            1.48  3            1.07
## bathrooms            2.27  1            1.51
## beds                 2.71  1            1.65
## accommodates         3.04  1            1.74

outlierTest(model3_box_BBB)

##       rstudent  unadjusted p-value    Bonferroni p
## 12543    -7.84 0.00000000000000476 0.0000000000717
## 10091    -7.34 0.00000000000023141 0.0000000034901
## 14352    -6.58 0.00000000004895500 0.0000007383400
## 5788      6.14 0.00000000084685000 0.0000127720000
## 14925     5.81 0.00000000630930000 0.0000951570000
## 11271     5.67 0.00000001484500000 0.0002239000000
## 12609     5.66 0.00000001560200000 0.0002353100000
## 9884     -5.64 0.00000001712500000 0.0002582700000
## 10193     5.30 0.00000012041000000 0.0018160000000
## 7687     -5.11 0.00000031792000000 0.0047949000000

Adding Bed, Bathrooms and bedrooms and size factors to the box cox model seemed to also dramatically improve the adjusted r-squared from ~27% to ~43%. All factors are significant at the 5% level with underlying equation assuming apartment property type as a base. We check the residual plots again which seems to show improvement in heteroscedasticity issues(as shown by the fitted vs residuals and Scale-Location plots) we also see again that listing 12543 falls outside of the 0.5 Cook’s distance and is an influential point that should be removed for model improvement. We also noted that bedrooms had a VIF greater than 5 and thus showed multi-collinearity issues, we therefore chose to remove it and run the model again which, this time, showed no multi-collinearity issues. Outliers were identified and examined.

Log model including host effects factors**

model4_log_hosts <-lm(log(price_4_nights)~prop_type_simplified+
                      number_of_reviews+
                      review_scores_rating+
                      room_type+bathrooms+
                      #bedrooms+
                      beds+
                      accommodates+host_is_superhost+ #added host effects
                        host_total_listings_count+
                        host_identity_verified, 
                    data=listings_model)


summary(model4_log_hosts)

## 
## Call:
## lm(formula = log(price_4_nights) ~ prop_type_simplified + number_of_reviews + 
##     review_scores_rating + room_type + bathrooms + beds + accommodates + 
##     host_is_superhost + host_total_listings_count + host_identity_verified, 
##     data = listings_model)
## 
## Residuals:
##    Min     1Q Median     3Q    Max 
## -5.277 -0.376 -0.059  0.313  5.813 
## 
## Coefficients:
##                                  Estimate Std. Error t value
## (Intercept)                      7.414781   0.065158  113.80
## prop_type_simplifiedGuest suite -0.204123   0.020471   -9.97
## prop_type_simplifiedHouse        0.006726   0.014203    0.47
## prop_type_simplifiedOther        0.077188   0.013970    5.53
## prop_type_simplifiedVilla        0.725077   0.030441   23.82
## number_of_reviews               -0.001807   0.000176  -10.24
## review_scores_rating             0.004590   0.000685    6.70
## room_typeHotel room              0.109130   0.036094    3.02
## room_typePrivate room           -0.348569   0.013433  -25.95
## room_typeShared room            -0.775051   0.060643  -12.78
## bathrooms                        0.150597   0.006324   23.82
## beds                            -0.069625   0.004333  -16.07
## accommodates                     0.164751   0.003765   43.76
## host_is_superhostTRUE            0.005984   0.012789    0.47
## host_total_listings_count        0.000844   0.000034   24.80
## host_identity_verifiedTRUE       0.086557   0.011840    7.31
##                                             Pr(>|t|)    
## (Intercept)                     < 0.0000000000000002 ***
## prop_type_simplifiedGuest suite < 0.0000000000000002 ***
## prop_type_simplifiedHouse                     0.6358    
## prop_type_simplifiedOther           0.00000003342798 ***
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews               < 0.0000000000000002 ***
## review_scores_rating                0.00000000002202 ***
## room_typeHotel room                           0.0025 ** 
## room_typePrivate room           < 0.0000000000000002 ***
## room_typeShared room            < 0.0000000000000002 ***
## bathrooms                       < 0.0000000000000002 ***
## beds                            < 0.0000000000000002 ***
## accommodates                    < 0.0000000000000002 ***
## host_is_superhostTRUE                         0.6399    
## host_total_listings_count       < 0.0000000000000002 ***
## host_identity_verifiedTRUE          0.00000000000028 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.611 on 15065 degrees of freedom
##   (83 observations deleted due to missingness)
## Multiple R-squared:  0.476,  Adjusted R-squared:  0.475 
## F-statistic:  912 on 15 and 15065 DF,  p-value: <0.0000000000000002

plot(model4_log_hosts)

hist(rstandard(model4_log_hosts))

bptest(model4_log_hosts)

## 
##  studentized Breusch-Pagan test
## 
## data:  model4_log_hosts
## BP = 1467, df = 15, p-value <0.0000000000000002

vif(model4_log_hosts)

##                           GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified      1.70  4            1.07
## number_of_reviews         1.23  1            1.11
## review_scores_rating      1.06  1            1.03
## room_type                 1.50  3            1.07
## bathrooms                 2.27  1            1.51
## beds                      2.71  1            1.65
## accommodates              3.04  1            1.74
## host_is_superhost         1.26  1            1.12
## host_total_listings_count 1.02  1            1.01
## host_identity_verified    1.06  1            1.03

outlierTest(model4_log_hosts)

##       rstudent         unadjusted p-value           Bonferroni p
## 12543    -9.66 0.000000000000000000000496 0.00000000000000000748
## 11271     9.55 0.000000000000000000001519 0.00000000000000002291
## 5788      9.29 0.000000000000000000017380 0.00000000000000026211
## 14925     8.64 0.000000000000000006036900 0.00000000000009104200
## 10091    -7.85 0.000000000000004267600000 0.00000000006436000000
## 12609     7.58 0.000000000000035536000000 0.00000000053592000000
## 14352    -7.57 0.000000000000039691000000 0.00000000059858000000
## 10193     6.20 0.000000000570840000000000 0.00000860890000000000
## 9329      6.17 0.000000000714730000000000 0.00001077900000000000
## 10830     6.17 0.000000000716870000000000 0.00001081100000000000

The inclusion of host characteristics again improved our adjusted-R squared (but not that significantly) from 45% to 47%. However both host identity verification and hosts total listings count were found to be significant at a 5% level. Being a super host had no significant effect on pricing and will thus be removed in further models.

Box cox model including host effects factors

model4_box_hosts <-lm(bcPower(price_4_nights,lambda = -0.25)~prop_type_simplified+
                      number_of_reviews+
                      review_scores_rating+
                      room_type+bathrooms+
                      #bedrooms+
                      beds+
                      accommodates+host_is_superhost+
                        host_total_listings_count+
                        host_identity_verified, 
                    data=listings_model)


summary(model4_box_hosts)

## 
## Call:
## lm(formula = bcPower(price_4_nights, lambda = -0.25) ~ prop_type_simplified + 
##     number_of_reviews + review_scores_rating + room_type + bathrooms + 
##     beds + accommodates + host_is_superhost + host_total_listings_count + 
##     host_identity_verified, data = listings_model)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -0.5080 -0.0424 -0.0022  0.0415  0.4361 
## 
## Coefficients:
##                                    Estimate  Std. Error t value
## (Intercept)                      3.39349548  0.00745510  455.19
## prop_type_simplifiedGuest suite -0.02718016  0.00234219  -11.60
## prop_type_simplifiedHouse       -0.00162996  0.00162508   -1.00
## prop_type_simplifiedOther        0.01003417  0.00159836    6.28
## prop_type_simplifiedVilla        0.05681804  0.00348287   16.31
## number_of_reviews               -0.00019683  0.00002019   -9.75
## review_scores_rating             0.00056181  0.00007841    7.16
## room_typeHotel room              0.00820489  0.00412965    1.99
## room_typePrivate room           -0.04717045  0.00153697  -30.69
## room_typeShared room            -0.11315680  0.00693844  -16.31
## bathrooms                        0.01393897  0.00072350   19.27
## beds                            -0.00660188  0.00049572  -13.32
## accommodates                     0.01818881  0.00043076   42.22
## host_is_superhostTRUE           -0.00057704  0.00146321   -0.39
## host_total_listings_count        0.00007397  0.00000389   19.00
## host_identity_verifiedTRUE       0.00956931  0.00135469    7.06
##                                             Pr(>|t|)    
## (Intercept)                     < 0.0000000000000002 ***
## prop_type_simplifiedGuest suite < 0.0000000000000002 ***
## prop_type_simplifiedHouse                      0.316    
## prop_type_simplifiedOther           0.00000000035283 ***
## prop_type_simplifiedVilla       < 0.0000000000000002 ***
## number_of_reviews               < 0.0000000000000002 ***
## review_scores_rating                0.00000000000082 ***
## room_typeHotel room                            0.047 *  
## room_typePrivate room           < 0.0000000000000002 ***
## room_typeShared room            < 0.0000000000000002 ***
## bathrooms                       < 0.0000000000000002 ***
## beds                            < 0.0000000000000002 ***
## accommodates                    < 0.0000000000000002 ***
## host_is_superhostTRUE                          0.693    
## host_total_listings_count       < 0.0000000000000002 ***
## host_identity_verifiedTRUE          0.00000000000169 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.0699 on 15065 degrees of freedom
##   (83 observations deleted due to missingness)
## Multiple R-squared:  0.447,  Adjusted R-squared:  0.446 
## F-statistic:  811 on 15 and 15065 DF,  p-value: <0.0000000000000002

plot(model4_box_hosts)

hist(rstandard(model4_box_hosts))

bptest(model4_box_hosts)

## 
##  studentized Breusch-Pagan test
## 
## data:  model4_box_hosts
## BP = 1224, df = 15, p-value <0.0000000000000002

vif(model4_box_hosts)

##                           GVIF Df GVIF^(1/(2*Df))
## prop_type_simplified      1.70  4            1.07
## number_of_reviews         1.23  1            1.11
## review_scores_rating      1.06  1            1.03
## room_type                 1.50  3            1.07
## bathrooms                 2.27  1            1.51
## beds                      2.71  1            1.65
## accommodates              3.04  1            1.74
## host_is_superhost         1.26  1            1.12
## host_total_listings_count 1.02  1            1.01
## host_identity_verified    1.06  1            1.03

outlierTest(model4_box_hosts)

##       rstudent  unadjusted p-value    Bonferroni p
## 12543    -7.84 0.00000000000000467 0.0000000000705
## 10091    -7.49 0.00000000000007202 0.0000000010862
## 14352    -6.62 0.00000000003811600 0.0000005748200
## 5788      6.25 0.00000000042417000 0.0000063969000
## 14925     5.90 0.00000000371120000 0.0000559690000
## 11271     5.76 0.00000000834710000 0.0001258800000
## 12609     5.74 0.00000000992930000 0.0001497400000
## 9884     -5.67 0.00000001437100000 0.0002167300000
## 10193     5.33 0.00000010059000000 0.0015170000000
## 7687     -5.18 0.00000022576000000 0.0034047000000

The inclusion of host characteristics again improved the adjusted R-squared from ~43% to ~45%.Being a super host as in the log transformed model again has no significant effect on price while host total listings and identity verification proved to be significant. At this point we also note that the box-cox transformation yielded weak adjusted r-squared values across the 4 models considered so far. We thus conclude the log transformed model as being a better transformation and continue our analysis only examining the log transformation of the dependent variable.